[Training] [0/n] Add preprocessing pipeline #442
Conversation
|
Need to merge #438 first, because this PR requires v1/datasets |
JerryZhou54
force-pushed
the
wei/preprocess
branch
2 times, most recently
from
93fd4ac to
5c3ec38
Compare
| local_dir=os.path.join( | ||
| 'data', BASE_MODEL_PATH)) |
There was a problem hiding this comment.
I'm not sure why we should use local_dir--this can make cache sharing more complicated. I've PRed to remove this
There was a problem hiding this comment.
it may have been me that added the hardcoded path? @JerryZhou54 perhaps lets just use model_path arg here and have the registry detect the correct pipeline config by directly using the HF string instead of a local path?
JerryZhou54
had a problem deploying
to
runpod-runners
GitHub Actions
Failure
| # export WANDB_MODE="offline" | ||
| GPU_NUM=1 # 2,4,8 | ||
| MODEL_PATH="Wan-AI/Wan2.1-T2V-1.3B-Diffusers" | ||
| TEXT_ENCODER_PATH="/Wan-AI/Wan2.1-T2V-1.3B-Diffusers/tokenizer" |
There was a problem hiding this comment.
instead of hardcoding this TEXT_ENCODER_PATH, we can simply do:
path = maybe_download_model(args.model_path)
encoder_path = os.join(path, 'tokenizer')
|
|
||
| logger = init_logger(__name__) | ||
|
|
||
| BASE_MODEL_PATH = "/workspace/data/Wan-AI/Wan2.1-T2V-1.3B-Diffusers" |
There was a problem hiding this comment.
let's just use args.model_path here instead?
|
feel free to merge after addressing my comments |
JerryZhou54
had a problem deploying
to
runpod-runners
GitHub Actions
Error
JerryZhou54
force-pushed
the
wei/preprocess
branch
from
3e9c40a to
31c34d0
Compare
JerryZhou54
temporarily deployed
to
runpod-runners
GitHub Actions
Inactive
| if __name__ == "__main__": | ||
| parser = argparse.ArgumentParser() | ||
| # dataset & dataloader | ||
| parser.add_argument("--model_path", type=str, default="data/mochi") |
There was a problem hiding this comment.
it's better to not use data except only for testers running on Runpod, just use hf's default cache path
Co-authored-by: “BrianChen1129” <[email protected]>
Uh oh!
There was an error while loading. Please reload this page.